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IDEAL framework for surgical innovation 1: the idea and 
development stages 
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behalf of the IDEAL group 

IDEAL is a framework for evaluations of surgical innova- 
tions, which follow a distinct development pathway dif- 
fering from the approach developed for pharmacological 
interventions. Many pathway and evaluation challenges 
are shared by other interventional therapies, requiring 
individual therapist skills and customisation of treatment 
to the individual, partly through medical devices. This 
paper provides an overview of the IDEAL framework and 
recommendations, and focuses on the first two stages: idea 
and development. 

Introduction 

Surgical innovations comprise new techniques, modified 
strategies, or innovative instruments. The evidence base 
for many of these approaches— and therefore for much of 
current surgical practice— is vastly weaker than for most 
modern drug treatments. Randomised trials of surgical 
techniques (versus placebo surgery) 1 were conducted 
within 10 years of the publication of the epochal strep- 
tomycin drug trial. 2 Yet, despite rapid growth in recent 
years, the overall number of randomised controlled trials 
and systematic reviews in surgical innovations remains 
small compared with the number of studies evaluating 
drug treatments. Randomised trials have also been few 
in number and of poor quality in some other therapeu- 
tic specialities, where the success of the intervention 
depends on the skill and judgment of the individual 
operator. 3 

The IDEAL Collaboration was born out of a series of con- 
ferences between surgeons and methodologists at Balliol 
College, Oxford/ 4 5 6 which was convened to study why 
high quality trials in surgery were genuinely difficult to 
conduct, and what could be done to improve the evidence 
base for surgery. The conclusion was that innovation in 
surgery inevitably follows a pathway with important dif- 



Box 1 1 Recommendations for studies in stages 1 (idea) and 
2 (development) 

Idea 

Mandatory registry for interventions thought to be first in man, 
with anonymous reporting option 
Protocols pre-registered on the above mandatory registry, for 
planned research programmes on a first-in-man intervention 
Development and use of agreed reporting standards and 
definitions for key outcomes and modifying factors 

Development 

Prospective development studies, with pre-published protocol 
and consecutive cases 

Publication of findings, including transparent reporting of 
changes in technique or device design and indication 

ferences from that followed by pharmacological devel- 
opments, and that a different approach to evaluation is 
therefore needed. It was noted that many non-surgical 
disciplines had similar problems with evaluation of such 
treatments (termed as "interventional therapies"), which 
rely on operator skill and tailoring of the intervention to the 
patient (for example, cardiac catheterisation, endoscopic 
techniques, or physiotherapy). 

The IDEAL Collaboration developed a framework for 
the stages in surgical innovation (idea, development, 
exploration, assessment, and long term study; table) and 
a set of recommendations on how evaluation should be 
conducted at each stage (box 1). The collaboration also 
proposed how the environment for surgical research 
could be improved by editors, regulators, funders, and 
professional societies. An open international collabora- 
tive group has been developed to explore these issues fur- 
ther.' Recent concerns over hip resurfacing techniques 8 
and breast implants 9 have raised serious questions about 
how medical devices are evaluated, and there has been 



IDEAL framework 


Stage 1: Idea 


Stage 2a: Development 


Stage 2b: Exploration 


Stage 3: Assessment 


Stage 4: Long term study 


Question 


Can the procedure or device achieve 
a specific physical or physiological 
goal? 


What is the optimal technique or 
design, and for which patients does 
it work best? 


What are the outcomes of more 
widespread use? Can consensus 
equipoise be reached on a trial 
question? 


How well does the procedure work 
compared with current standards 
of care? 


What are the long term effects and 
outcomes of the procedure? 


Aim 










Proof of concept 


Safety, efficacy 


Efficacy 


Comparative effectiveness 


Quality assurance 


Patient base 










Single to few 


10s 


100s 


100s+ 


100s+ 


Optimal study design(s) 










First-in-man study; structured case 
report 


Prospective development study 


Prospective collaborative 
observational study (Phase IIS) or 
feasibility randomised controlled trial 
(or both) 


Randomised controlled trial 


Observational study or randomised 
trial nested within a comprehensive 
disease based registry 


Example of procedure at this stage 










Stem cell based tracheal transplant 
for tracheal stenosis 2 


Peroral endoscopic myotomy for 
oesophageal achalasia 


Single incision laparoscopy for 
abdominal surgery 


Minimally invasive oesophagectomy 


Banding and bypass surgery for 
morbid obesity 



BMJ RESEARCH METHODS AND REPORTING 



1of4 



RESEARCH METHODS AND REPORTING 



Box 2 1 Example of study at idea stage 1 

Stem cell tracheal transplant (based on reference 1 1) 

Clinical background at the time of conduct 

Loss of an airway is debilitating and replacement is difficult 

A stem cell graft embedded in a framework constructed using a de-antigenised collagen matrix may 

overcome current limitations 

Design 

Staged programme of multidisciplinary research beginning with detailed preclinical studies 
Explicit clinical and data collection protocol written in advance and submitted for ethical review 
Detailed informed consent process 
Oversight provided by multiple agencies 
Findings 

Replacement using stem cell tracheal graft is feasible and a good early outcome is achievable 
No sign of rejection was achieved without the need for immunosuppressive drugs 

considerable interest in applying the IDEAL framework 
to this problem, since many difficulties in evaluating 
device innovation mirror those in surgery innovation. In 
this series of three articles, we explain the problems and 
discuss proposed solutions put forward in the IDEAL rec- 
ommendations, using current examples. This first article 
in the series focuses on the first two stages of the IDEAL 
framework: idea and development. 

Idea (IDEAL stage 1) 

Surgical innovations can arise from careful planning and 
laboratory studies, from necessity created by an emer- 
gency, or even by accident. Advances in technology and 
related devices may make new or substantially different 
procedures feasible (such as robotic surgery). Planned and 
unplanned innovations can also occur out of desperation, 
in situations where the prognosis seems otherwise hope- 
less (for example, abbreviated "damage control" surgery 
for major combined vascular and visceral injury 10 ). More 
measured innovation could represent an incremental 
advance, where the new procedure is a small variant on 
an older one. Innovations can also be completely novel, 
and be taken through to clinical trials via a carefully 
planned research programme, such as the recent success- 
ful advances in transplant surgery. 11 

What should a surgeon do if they believe that they have 
invented or developed something new and different, or 
if— in the case of industry driven research— they have used 
a new device in humans for the first time? The answer 
has two parts: surgeons can report what they have done, 
and then evaluate the intervention. Because first-in-man 
studies, by their nature, deal with single cases or a small 
number of cases, study design considerations might be 
largely irrelevant; but how they are reported is impor- 
tant. We can develop basic principles using the three pil- 
lars of the modern framework of medical ethics: utility, 
beneficence, and non-maleficence. 12 Surgeons should 
have the opportunity of learning from each other's expe- 
riences, particularly if this helps their patients or avoids 
them being harmed. Therefore, surgeons have an ethical 
obligation to share experiences with colleagues. Further, 
they need to convey sufficient information about what 
was done and what the consequences were, so that spe- 
cialist colleagues can understand how to reproduce their 
success or avoid their failure. 



All first-in-man interventions (whether a new proce- 
dure or new use of a device) should be included in an 
open access registry recording key details of the innova- 
tion. This registry would facilitate information searches 
in the published literature, which surgeons should carry 
out before embarking on a planned first-in-man interven- 
tion. 1 3 Box 2 provides an example of a study at the idea 
stage. The use of a new innovation, particularly if it is the 
first use of the innovation at the surgeon's institution, 
should have some form of independent oversight by those 
responsible for local clinical governance. Consent for new 
procedures is important: patients contemplating whether 
to undergo such procedures must fully understand their 
experimental nature, and the uncertainty that therefore 
surrounds any estimates of risk. If patient incapacity or 
time urgency prevents informed consent, governance 
authorities and patients' relatives or advocates may need 
to reach an agreement by discussion, even for retrospec- 
tive cases. Therefore, hospitals would need systems that 
allow the right mix of clinical and ethical expertise to be 
bought to bear rapidly, and outside of normal hours if 
necessary. 

Although surgeons may not need much incentive to 
report their successful innovations, it is arguably just 
as important to formally record their unsuccessful ideas 
or initial failures, to avoid unnecessary repetition by 
others. For this reason, the IDEAL recommendations 
include registration of all first-in-man procedures, with 
the suggestion that anonymous reporting might be per- 
mitted. In principle, anonymous reporting of harms or 
"near misses" might be desirable, but it has serious ethi- 
cal, practical, and legal difficulties. If reporting is truly 
anonymous, how can spam or deliberately misleading 
reports be screened out? On the other hand, if identifi- 
cation of the author is possible in principle, legal dis- 
covery attempts and claims for compensation are a near 
certainty. The unique nature of new procedures might 
also make it difficult to maintain patient confidentiality. 
To allow surgeons to report their unsuccessful first-in- 
man efforts with confidence, a legal framework may be 
required, supported by the relevant governance and pro- 
fessional bodies to protect surgeons from compensation 
claims, provided that oversight and informed consent 
have been satisfactory. 

Development (IDEAL stage 2a) 

The IDEAL development stage begins once surgeons start 
to plan a series of procedures using a new technique or 
device (table). Innovations are especially fluid in this 
phase; innovations undergo rapid iterative change in the 
light of experience. Therefore, it is the development stage 
that most clearly differentiates the pathway for surgery 
innovation from that for pharmaceutical innovations. In 
both a scientific and ethical sense, development is the 
most problematic of the stages, and as a result is often 
poorly reported. 

Experience often makes the need for modification obvi- 
ous after only a few repetitions, although surgeons are 
insecure about the logical and ethical justification for 
making changes on the basis of scarce data that are not 
definitive. It is therefore tempting for authors to wait until 
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Box 3 1 Common items for which agreed standard 
definitions are needed 

Contextual factors 

Grading of patient risk factors 

Severity grading of comorbid pathology or general health 

Scale of surgical insult 

Urgency status of procedure 

Environment for surgery (hospital or unit type) 

Outcomes 

Grading of functional performance 
Scope and severity of complications 

the development stage has ended, and then report the 
initial results as if the final version of the technique had 
been used in all cases. This strategy is adopted in the clas- 
sic retrospective surgical case series, and is deeply prob- 
lematic. Obscuring details that authors may not wish to 
report deprives others of the opportunity to learn from the 
development process, and can provide a misleading pic- 
ture of the use of a procedure or device. Authors obscur- 
ing changes in eligibility, which naturally occur during 
development, in order to make it appear predetermined 
is similarly unhelpful. If the patients undergoing a tech- 
nique are different at the beginning and end of a reported 
series, the aggregate outcome might not indicate much 
about what can be expected from using the final version 
of the technique in the patient group that trial and error 
has shown to be most suited to it. 

Judgments about success or failure at this stage may be 
made on the basis of short term outcome measures that 
might not reflect the most important effects of the proce- 
dure, and frequently the data are insufficient to allow any 
meaningful statistical analysis (even if available data are 
maximised 17 '). A cancer operation or a new artificial joint 
might seem successful at this stage because recovery from 
the surgery is quick or complications few, but subsequent 
data about survival rates or function in the long term may 
reverse these impressions. One may reasonably question 
the value of reporting such unreliable figures, but the 
alternative may be waiting many years for the results of 
definitive trials— which are unlikely to be undertaken 
without some pilot data. The pressure to innovate and 
improve is such that funders, patients, and clinical col- 
leagues expect to be updated on the promise of innova- 
tions as rapidly as possible, in order to make decisions 
about funding, treatment, or use. If such decisions are to 
be made regardless (which seems a reasonable assump- 
tion barring a radical change in how health provision is 
organised internationally), they should at least be made 
using the most complete and accurate information avail- 
able. The key principle, therefore, is transparency. 

The IDEAL recommendations recognise that at the 
development stage, a randomised trial is often operation- 
ally undesirable and scientifically of limited use, owing 
to procedural modifications and varying eligibility. IDEAL 
supports prospective rather than retrospective studies at 
this stage, with sequential reporting of all cases and out- 
comes without omissions, and with clear explanations 
of when and how technique, design, or indications were 
changed. Sequential presentation of results might also 



reveal the effects of operator learning curves, which have 
an even more important role in the next stage of explo- 
ration. To ensure full reporting of relevant outcomes, the 
prior publication of a protocol at the outset of this type of 
study would be helpful. The United States Food and Drug 
Administration, which has been re-evaluating the regula- 
tory framework for implantable devices since the Institute 
of Medicine report of 201 1, 15 has put forward proposals for 
early studies of innovative devices that closely follow this 
model, which is encouraging. 

The prospective development studies recommended by 
the IDEAL Collaboration represent a new type of obser- 
vational study, which will no doubt change and evolve, 
but examples of this kind of study are now beginning 
to appear. 16 17 Key elements are a prior protocol, clearly 
defined objective outcomes, and transparent sequential 
reporting of cases, showing when changes in indication 
or technique are made. Data from this type of study will 
be more reliable and valid than information obtained from 
retrospective series, although retrospective data require 
much less effort and planning. We therefore suggest that 
for techniques and devices in the development stage, 
journals positively discriminate in favour of prospective 
studies, and should cease to accept studies based on ret- 
rospective data except when it can convincingly be shown 
that no viable alternative exists. 

A much needed, important parallel improvement is the 
development of international standards for reporting sur- 
gical outcomes and contextual factors. Reports that use a 
common terminology and taxonomy are much more useful 
than those in which a plethora of definitions of the key 
data sow confusion and doubt. Groups such as COMET 18 
and the Zurich group responsible for the Dindo-Clavien 
classification of complications 19 have made an important 
contribution to standardising this language, but further 
work is still needed. Many specialist endpoints will be best 
defined by consensus among the specialist community, 
and specialist societies and journals should work together 
to standardise terminology in their area of interest. Box 3 
shows key outcomes and contextual factors that will need 
general agreement across the international surgical com- 
munity. This agreement will need a concerted effort from 
international societies, national professional bodies, and 
leading journals, but research funders could also help 
by insisting on the use of standardised terms in funding 
applications. 

Discussion 

Early evaluations of a surgical innovation face common 
challenges; however, these difficulties must not prevent 
such studies being conducted. Current practices of study 
design and reporting are suboptimal and need upgrading. 
In particular, meaningful reporting of first-in-man cases 
should become routine (irrespective of the findings). Stud- 
ies in the development stage need to be prospective, based 
on consecutive case reporting, and need to be open about 
the changes in indication, technique, and use of equip- 
ment that occur as experience is gained. Studies in the 
development stage also need a rapid, flexible, and expert 
system of governance to make decisions about whether 
to permit new procedures or devices to go ahead, and 



BMJ RESEARCH METHODS AND REPORTING 



3 of 4 



RESEARCH METHODS AND REPORTING 



SUMMARY POINTS 

Innovations in surgery have several features that make scientific 
evaluation challenging, such as an early phase of rapid 
modification, learning curve, and strong therapist preferences 
The IDEAL framework describes five stages of development and 
evaluation for surgical and interventional innovations: idea, 
development, exploration, assessment, and long term study 
The IDEAL recommendations identify design and reporting 
ideas that could help in dealing with specific problems at each 
stage in the framework 

At stage 1 (idea), accuracy, transparency, and completeness 
of reporting are key elements. Recommendations include 
standardisation of reports and development of an open access 
database for lodging reports of first-in-man procedures 
At stage 2a (development), innovations are in a state of 
flux, undergoing modifications and changes in indication. 
Prospective studies with comprehensive sequential reporting 
of changes to technique and indication are recommended, 
together with standardisation of terminology 

particularly about whether these new interventions can 
be modified, as is typical during this stage. Research eth- 
ics committees and device regulatory bodies could help 
by requiring a declaration of the IDEAL stage that the 
investigators feel the device or procedure has reached, 
with supporting evidence. Innovations in the idea stage 
would then be expected to lead to proposals for a prospec- 
tive development study. 

Modifications of interventions are a problem because of 
their likely frequency and the need for a rapid and ethical 
response. This task could be delegated to hospitals and 
universities, because more centralised bodies would be 
unable to gather information and respond appropriately 
in a realistic timescale. It would be sensible for existing 
structures to take on this role, in addition to their other 
functions, rather than to set up a new infrastructure. In the 
United Kingdom, trusts each have a committee to review 
proposals for procedures that are new to the trust, which 
would be the obvious body to prepare such a response. 
It is essential that the body responsible for providing an 
ethical opinion abides by certain principles to maintain 
an appropriate balance between fostering innovation and 
protecting current patients. Such considerations include 
developing sensible standards for documentation, report- 
ing, and patient consent procedures; ensuring oversight of 
the committee itself; and ensuring access to suitable expert 
advice. Professional societies and bodies, healthcare insti- 
tutions, and national regulatory agencies can all contribute 
towards better surgical research. 

Summary 

Early evaluations of a surgical innovation (whether an oper- 
ation, invasive procedure, or use of a medical device) face a 
common set of difficulties, related principally to the need to 
modify and redefine the intervention and indication during 
evaluation. The IDEAL recommendations propose abandon- 
ment of retrospective case series, and instead recommend 
adoption of mandatory registration of first-in-man reporting, 
standardisation of reporting, and use of prospective study 
designs. Current regulatory and ethical governance struc- 
tures need to be refined to facilitate an appropriate balance 
between fostering innovation and protecting patients. 
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